1. Field of the Invention
The present inventions relate generally to methods and systems for efficient streaming of messages from a source queue to a destination queue.
2. Description of the Prior Art and Related Information
Applications may communicate with each other via the transmission and receipt of messages. For example, a message may be transmitted from one queue in one database and received in another queue in another database. Message propagation involves sending data between source and destination queues. Messages are conventionally dequeued from a source queue one by one and enqueued in a destination queue one by one. When the destination queue resides on a different node, data must be encapsulated in a remote procedure call (RPC) that executes on the destination node. RPC is a powerful technique for constructing distributed applications. By using remote procedure calls, programmers of distributed applications avoid the details of the interface with the network, as the RPC isolates the application from the physical and logical elements of the data communications mechanism and allows the application to use a variety of transports. An RPC is similar to a function call. Like a function call, when an RPC is made, the calling arguments are passed to the remote procedure and the caller waits for a response to be returned from the remote procedure. The client makes a procedure call that sends a request to the server and waits. When the request arrives, the server calls a dispatch routine that performs the requested service, and sends the reply to the client. After the RPC call is completed, the client program continues.
FIG. 1 shows the structure of a conventional RPC. As shown, the remote procedure call 100 includes at least metadata 102, a function name 104, in arguments 106 to the function identified by the function name 104 and out arguments 108 returned by the function name 104. The metadata 102 represents overhead to the RPC 100, whereas the function name 104, the in argument 106 and the out argument 108 forms part of the payload of the RPC 100. It may be appreciated that, if the metadata is the same across several RPCs, repeatedly sending the same metadata to the destination side is redundant and inefficient.
FIG. 2 shows a conventional method of sending a plurality of RPCs between source queues and destination queues. As shown, a first application 202 sends a plurality of messages from a source queue 206 to destination queues 208, 210 accessible to a second application 204 (the ultimate consumer of the messages). The applications may reside on the same or different databases. Each of the messages may be successively dequeued from the source queue Q1 206, encapsulated into separate RPCs by a propagator process (not shown), and sent to its destination queue or queues. In FIG. 2, the destination for RPC 212, 214 and 216 is Q2 208, whereas the destination for RPC 218 and RPC 220 is Q3 210. Conventionally, each message is encapsulated within a separate RPC, as shown at 212-218. After executing the function called for by the RPC, the destination becomes the source and the result of the RPC is called back to the original source of the RPC, as indicated in FIG. 2 by the arrows from the destination back to the source.
As implicit in FIG. 2, propagating a batch of n messages conventionally requires the formation, sending and execution of n RPCs. That is, conventional messaging techniques and propagation architectures require a separate RPC for each message to be enqueued at the destination queue in the same or another database. It may be appreciated that sending n separate RPCs results in an inefficient use of system resources. Basic system resources include the client, the network and the server. When a client prepares a single RPC, the server and the network connection between the client and server sit idle. When the client sends the RPC over the network, the client and server sit idle. When the server processes the RPC, the client and the network connection between the client and server sit idle. Indeed, associated with each message that is sent to its destination, a call stack is executed to carry out the enqueue operation and this call stack is re-executed for each message that is to be enqueued at the destination. When the response is transmitted to the client over the network, both the server and client sit idle. As a result of these serial inefficiencies, the throughput of the system (that is, the number of requests for service processed in a given amount of time) is low. Moreover, if multiple queues in the target database are each to receive the message, a separate copy of the message (one RPC per message enqueued) must be sent to each of the multiple queues in the target database. A side effect of executing multiple RPCs is reduced data sharing, in that messages and the messages' overhead that are to be propagated to multiple destination queues must be transmitted repeatedly, as conventional messaging schemes do not provide for any mechanism to share data between remote message enqueues. For example, when multiple messages are propagated to the same destination queue, the message's metadata is transmitted repeatedly. For persistent queue propagation, each RPC translates into a separate Structured Query Language (SQL) insert statement, each of which consumes limited CPU resources at the destination side.
From the foregoing, it is clear that methods and systems are needed to more efficiently send data between source and destination queues. More specifically, what are needed are methods and systems for more efficiently sending multiple messages from a source queue to a destination queue. Also needed are methods and systems to more efficiently propagate messages to multiple destination queues. That is, what are needed are methods and systems that do not require each message to be encapsulated in a separate RPC for propagation from a source queue to one or more destination queues.